Multimedia at Work Editor : Tiziana Catarci University of Rome

tem provides integrated support for spatiotemporal and semantic queries for video.1 A knowledge base—consisting of a fact base and a comprehensive rule set implemented in Prolog— handles spatio-temporal queries. These queries contain any combination of conditions related to direction, topology, 3D relationships, object appearance, trajectory projection, and similaritybased object trajectories. The rules in the knowledge base significantly reduce the number of facts representing the spatio-temporal relations that the system needs to store. A feature database stored in an object-relational database management system handles semantic queries. To respond to user queries containing both spatio-temporal and semantic conditions, a query processor interacts with the knowledge base and object-relational database and integrates the results returned from these two system components. Because of space limitations, here we only discuss the Web-based visual query interface and its fact-extractor and video-annotator tools. These tools populate the system’s fact base and feature database to support both query types. System architecture We built BilVideo over a client–server architecture, shown in Figure 1. Users access BilVideo over the Internet through a Java client applet. The “Query Types” sidebar discusses the forms of user queries that BilVideo’s architecture supports. The heart of the system is the query processor, which runs in a multithreaded environment. The query processor communicates with a feature database and the knowledge base, where the system stores semantic and fact-based metadata, respectively. The system stores raw video data and its features in a separate database. The feature database contains video semantic properties to support keyword-, activity/event-, and categorybased queries. The video-annotator tool, which we developed as a Java application, generates and maintains the features. As mentioned previously, the knowledge base supports spatio-temporal queries and the facts base is populated by the factextractor tool, which is also a Java application.